inspector,http: support builtin http request bodies by GrinZero · Pull Request #62915 · nodejs/node

GrinZero · 2026-04-23T17:54:50Z

Summary

This PR adds builtin http/https request-body support to network inspection, so Network.getRequestPostData can return text request bodies while preserving the existing rejection behavior for binary request bodies.

It also moves builtin http response-body tracking to a raw-byte hook before IncomingMessage decoding, so response inspection remains correct even when user code calls response.setEncoding(...).

In addition, this PR extends requestWillBeSent to accept initiator data from JS diagnostics payloads and validates structured initiator.stack objects against the inspector protocol schema before forwarding them to DevTools.

Problem

Builtin http/https network inspection currently emits:

Network.requestWillBeSent
Network.responseReceived
Network.loadingFinished

However, the builtin http client path does not currently expose request-body data to the inspector. As a result, Network.getRequestPostData cannot return POST data for builtin http/https requests.

There are two related gaps as well:

IncomingMessage 'data' events are not a stable source of raw bytes. If user code calls response.setEncoding('utf8'), the chunks observed by userland become strings, while the inspector protocol expects byte-oriented payloads.
requestWillBeSent needs to accept JS-provided initiator data, but structured stack trace input must be validated before it is forwarded as protocol data.

Approach

1. Reuse the existing request buffering pipeline

This change does not alter the CDP schema or the existing buffering behavior in NetworkAgent.

Instead, it reuses the current pipeline already used by other transports:

Network.dataSent(...) -> NetworkAgent::getRequestPostData(...)

2. Add builtin `http` request-body diagnostics events

The builtin http client now publishes:

http.client.request.bodyChunkSent
http.client.request.bodySent

These events are emitted from the ClientRequest write path before HTTP framing is applied, so the inspector sees the original user payload rather than chunked-transfer framing bytes.

3. Capture builtin `http` response bytes before decoding

For responses, this PR avoids relying on IncomingMessage.on('data') in network_http.js.

Instead, it adds:

http.client.response.bodyChunkReceived

This hook runs before Readable.setEncoding() transforms chunks for userland, so the inspector always receives raw bytes.

4. Support and validate network initiator payloads

Network.requestWillBeSent can now use JS diagnostics payloads to provide initiator data.

On the C++ side, the incoming JS object is converted into inspector protocol values, and initiator.stack is validated against the Runtime.StackTrace schema before being attached to the event. If the initiator payload is malformed, the event is rejected with a clear error instead of forwarding bad protocol data to the frontend.

If JS does not provide an initiator, the existing behavior remains unchanged: the inspector captures the current stack trace and emits a default script initiator.

5. Document the new diagnostics channels

This PR also updates diagnostics_channel docs to cover the new builtin HTTP client request/response body channels and their message shapes.

Behavior

After this change:

builtin http and https POST requests with UTF-8 text bodies are available through Network.getRequestPostData
binary request bodies still reject with the existing inspector error behavior
builtin http response inspection continues to work even if user code calls response.setEncoding('utf8')
Network.requestWillBeSent can carry JS-provided initiator stack data
malformed initiator stack input is rejected by C++ schema validation

Tests

This PR adds and extends coverage in:

test/parallel/test-diagnostics-channel-http.js
test/parallel/test-inspector-network-http.js
test/parallel/test-inspector-network-arbitrary-data.js
test/parallel/test-inspector-emit-protocol-event-errors.js

The updated tests cover:

request body chunk and request body finished diagnostics events
response body raw-byte diagnostics events
text request bodies split across write() and end()
binary request bodies
http and https Network.getRequestPostData
binary request-body rejection semantics
response inspection when the client calls response.setEncoding('utf8')
JS-provided initiator stack parsing and forwarding
malformed initiator stack error handling

Verification

Verified locally with:

python3 tools/test.py \
  parallel/test-diagnostics-channel-http \
  parallel/test-inspector-network-http \
  parallel/test-inspector-network-arbitrary-data \
  parallel/test-inspector-emit-protocol-event-errors

Refs

Tracking issue: Stabilizing network inspection in Node.js #53946

nodejs-github-bot · 2026-04-23T17:54:55Z

Review requested:

@nodejs/http
@nodejs/inspector
@nodejs/net

GrinZero · 2026-04-23T17:56:01Z

E2E Test:

const http = require("http");

const DEFAULT_TARGET_URL = "http://jsonplaceholder.typicode.com/posts";
const DEFAULT_PORT = Number(process.env.PORT || 3000);
const DEFAULT_HOST = "127.0.0.1";
const targetUrl = process.argv[2] || process.env.TARGET_URL || DEFAULT_TARGET_URL;
const defaultPayload = {
  title: "node inspect demo",
  body: "post payload from local trigger service",
  userId: 123,
};

if (!targetUrl.startsWith("http://")) {
  console.error(
    `[config] This script only uses the http module for outbound requests. Use an http:// URL, got: ${targetUrl}`
  );
  process.exit(1);
}

function readRequestBody(req) {
  return new Promise((resolve, reject) => {
    let body = "";

    req.setEncoding("utf8");
    req.on("data", (chunk) => {
      body += chunk;
    });
    req.on("end", () => {
      resolve(body);
    });
    req.on("error", reject);
  });
}

function buildPayload(rawBody) {
  if (!rawBody || !rawBody.trim()) {
    return defaultPayload;
  }

  const parsed = JSON.parse(rawBody);
  if (!parsed || Array.isArray(parsed) || typeof parsed !== "object") {
    throw new Error("payload must be a JSON object");
  }

  return {
    ...defaultPayload,
    ...parsed,
  };
}

function sendOutboundPost(url, payload) {
  return new Promise((resolve, reject) => {
    const body = JSON.stringify(payload);

    console.log(`\n[outbound] request -> POST ${url}`);
    console.log(`[outbound] request payload: ${body}`);

    const outboundReq = http.request(
      url,
      {
        method: "POST",
        headers: {
          "user-agent": "node-inspect-http-demo",
          accept: "application/json,text/plain,*/*",
          "content-type": "application/json; charset=utf-8",
          "content-length": Buffer.byteLength(body),
        },
      },
      (outboundRes) => {
        let responseBody = "";
        outboundRes.setEncoding("utf8");

        console.log(
          `[outbound] response <- ${outboundRes.statusCode} ${outboundRes.statusMessage || ""}`.trim()
        );
        console.log("[outbound] response headers:", outboundRes.headers);

        outboundRes.on("data", (chunk) => {
          responseBody += chunk;
        });

        outboundRes.on("end", () => {
          console.log("[outbound] body preview:");
          console.log(responseBody.slice(0, 300) || "<empty>");

          resolve({
            statusCode: outboundRes.statusCode,
            statusMessage: outboundRes.statusMessage,
            headers: outboundRes.headers,
            bodyPreview: responseBody.slice(0, 300),
          });
        });
      }
    );

    outboundReq.on("socket", (socket) => {
      socket.on("connect", () => {
        console.log(
          `[outbound] socket connected -> ${socket.remoteAddress}:${socket.remotePort}`
        );
      });
    });

    outboundReq.on("finish", () => {
      console.log("[outbound] request finished");
    });

    outboundReq.on("error", (error) => {
      console.error("[outbound] request error:", error.message);
      reject(error);
    });

    outboundReq.write(body);
    outboundReq.end();
  });
}

function writeJson(res, statusCode, payload) {
  const body = JSON.stringify(payload, null, 2);
  res.writeHead(statusCode, {
    "content-type": "application/json; charset=utf-8",
    "content-length": Buffer.byteLength(body),
  });
  res.end(body);
}

const server = http.createServer(async (req, res) => {
  const url = new URL(req.url || "/", `http://${req.headers.host || "127.0.0.1"}`);

  if (req.method === "GET" && url.pathname === "/") {
    return writeJson(res, 200, {
      ok: true,
      message: "local trigger service is running",
      endpoints: {
        trigger: "POST /trigger",
      },
      targetUrl,
      defaultPayload,
    });
  }

  if (req.method === "POST" && url.pathname === "/trigger") {
    console.log(`\n[inbound] trigger <- ${req.method} ${url.pathname}`);

    try {
      const rawBody = await readRequestBody(req);
      const payload = buildPayload(rawBody);
      const outbound = await sendOutboundPost(targetUrl, payload);

      return writeJson(res, 200, {
        ok: true,
        targetUrl,
        payload,
        outbound,
      });
    } catch (error) {
      return writeJson(res, 500, {
        ok: false,
        error: error.message,
      });
    }
  }

  return writeJson(res, 404, {
    ok: false,
    error: "not found",
  });
});

server.listen(DEFAULT_PORT, DEFAULT_HOST, () => {
  console.log(`[server] listening on http://${DEFAULT_HOST}:${DEFAULT_PORT}`);
  console.log(`[server] outbound target: ${targetUrl}`);
  console.log("[usage] node --inspect-wait --experimental-network-inspection inspect-http.js");
  console.log(
    `[usage] curl -X POST http://${DEFAULT_HOST}:${DEFAULT_PORT}/trigger -H 'content-type: application/json' -d '{"title":"manual trigger"}'`
  );
});

metcoder95

lgtm, it just seems that is gonna need a documentation for the new dcs

GrinZero · 2026-04-24T10:38:17Z

Thanks for the feedback. I’ve updated the documentation in doc/api/diagnostics_channel.md to address the review comments and include the new HTTP diagnostics channel events.

Specifically, I added/updated entries for:

http.client.request.bodyChunkSent
http.client.request.bodySent
http.client.response.bodyChunkReceived

I also aligned the wording and structure with the existing Node.js docs style for built-in channels.

GrinZero · 2026-04-27T01:03:06Z

A slight push, hoping that a maintainer will notice.

GrinZero · 2026-04-27T09:19:20Z

Sorry for the ping, @legendecas

If you have some time, could you please help review this PR and trigger CI?

This PR fixes an issue where the network inspector does not show postData for http / https requests.

Thanks a lot!

GrinZero · 2026-04-28T07:10:33Z

push, request-ci plz

marco-ippolito · 2026-04-28T07:20:54Z

Ill try to review the PR later, I think it's important we make sure it doesn't leak memory or keep the body buffer reference alive, which can exhaust resource quickly @ShogunPanda ptal

GrinZero · 2026-04-28T07:23:52Z

Ill try to review the PR later, I think it's important we make sure it doesn't leak memory or keep the body buffer reference alive, which can exhaust resource quickly @ShogunPanda ptal

I'll go explore the issue you mentioned

GrinZero · 2026-04-28T07:55:49Z

Thanks for raising this concern.

I did a focused memory/lifetime check for this PR, mainly around two questions:

whether the inspector keeps the original userland request/response body buffers alive;
whether the new body buffering path can grow without bound.

Based on the current implementation and local testing, I do not see evidence that this PR keeps the original JS buffers alive.

For both request and response bodies, the payload bytes are copied into inspector-owned storage before being retained by NetworkAgent. I also verified this with a WeakRef/GC test: the original 4 MiB request body buffer and the observed response chunk both became collectible after the request completed, while Network.getRequestPostData() / Network.getResponseBody() could still return the payloads.

So the inspector is retaining copied payload bytes, not the original userland Buffer objects.

I also ran a repeated-request memory test with explicit Network.enable() limits. Memory increased initially, as expected from inspector-side buffering, but then reached a plateau and older requests were evicted. That behavior looks consistent with bounded retention by the existing request buffer manager, not an unbounded leak.

GrinZero · 2026-04-28T08:00:09Z

The test plan above was proposed by AI, and I reviewed it.

For the first test, it uses WeakRef, manual GC, and process.memoryUsage() to verify object lifetime. I think the underlying testing principle is reasonably credible.

The second test is a stress test. It lowers maxTotalBufferSize and maxResourceBufferSize, then sends multiple requests with relatively large bodies. I’m less familiar with this approach.

GrinZero · 2026-04-29T05:29:43Z

I merged the main branch code to avoid CI errors.

ShogunPanda

LGTM!

GrinZero · 2026-04-30T18:16:56Z

LGTM!

Thanks 🙏 I think we should add the request-ci tag to trigger the next steps. I have synced the code of the main branch

codecov · 2026-05-03T05:49:48Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.05%. Comparing base (bbf51ad) to head (d0f603a).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #62915      +/-   ##
==========================================
+ Coverage   90.03%   90.05%   +0.01%     
==========================================
  Files         713      713              
  Lines      224950   225186     +236     
  Branches    42532    42590      +58     
==========================================
+ Hits       202542   202790     +248     
+ Misses      14175    14174       -1     
+ Partials     8233     8222      -11

Files with missing lines	Coverage Δ
lib/_http_client.js	`97.44% <100.00%> (+<0.01%)`	⬆️
lib/_http_common.js	`100.00% <100.00%> (ø)`
lib/_http_outgoing.js	`96.20% <100.00%> (+0.07%)`	⬆️
lib/internal/http.js	`92.14% <100.00%> (+0.02%)`	⬆️
lib/internal/inspector/network_http.js	`97.35% <100.00%> (+1.67%)`	⬆️
src/inspector/network_agent.cc	`90.50% <100.00%> (+2.46%)`	⬆️

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

GrinZero · 2026-05-05T10:26:50Z

What are the commits after 98830e2 doing?:

Root cause

onClientRequestCreated was emitting Network.requestWillBeSent at request-creation time and computing:

hasPostData: !request.writableEnded

At that moment, writableEnded is always false, so DevTools always saw hasPostData: true — even for body-less calls like http.get().

The fix

Defer the requestWillBeSent emission until body state is actually known:

bodyChunkSent arrives → hasPostData = true
request.start / error / responseFinish reached without any chunk → hasPostData = false

A kRequestWillBeSent symbol guards against double-emission.

Why the C++ side has to change

The previous network_agent.cc::requestWillBeSent filled in the initiator like this:

.setStack(v8_inspector_->captureStackTrace(true)->buildInspectorObject(0))

That relied on an implicit invariant: when JS calls Network.requestWillBeSent, the JS stack is the user's stack, because the emission happens synchronously inside the http.get() call chain.

Once the emission is deferred, the trigger point becomes a diagnostics_channel or socket callback, and a stack captured in C++ at that moment points into Node.js internals — the initiator becomes useless.

The only correct fix is to capture the stack on the JS side at request-creation time via createInitiator() + getStructuredStack, stash it on the request, and pass it through when the actual emission happens.

That in turn requires the C++ backend to accept a JS-supplied initiator object, which is why:

createInitiatorFromObject was added to parse { type, stack, url, lineNumber, ... }.
A generic V8ToProtocolValue was added to convert the V8 stack object to protocol::Value, then rehydrated into a protocol StackTrace via ValueConversions<Runtime::API::StackTrace>::fromValue.
requestWillBeSent now uses the JS-supplied initiator when present and falls back to captureStackTrace otherwise, preserving backward compatibility.

How the follow-up commits fit in

reject malformed initiator stack traces, document network initiator validation, drop unreachable initiator.stack schema, tighten createInitiator coverage, etc., are all hardening and documentation for the new JS → C++ initiator path introduced above — not independent refactors.

GrinZero · 2026-05-06T08:23:49Z

Ill try to review the PR later, I think it's important we make sure it doesn't leak memory or keep the body buffer reference alive, which can exhaust resource quickly @ShogunPanda ptal

Gently push forward the progress

GrinZero · 2026-05-08T16:03:37Z

PUSH

Signed-off-by: GrinZero <774933704@qq.com>

GrinZero · 2026-05-09T06:32:25Z

Push the review slightly, the next plan is to promote websocket native, websocket from http, and SSE native, this part of the bug fix will be more important than the basics, if it can be completed first, the other PRs are not prone to conflicts

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Apr 23, 2026

metcoder95 approved these changes Apr 24, 2026

View reviewed changes

GrinZero force-pushed the fix-http-inspector-request-body branch from ab906d5 to 194a5fd Compare April 29, 2026 05:28

legendecas added inspector Issues and PRs related to the V8 inspector protocol http Issues or PRs related to the http subsystem. labels Apr 29, 2026

GrinZero mentioned this pull request Apr 29, 2026

http,inspector: add WebSocket upgrade observability #62933

Open

ShogunPanda approved these changes Apr 30, 2026

View reviewed changes

GrinZero force-pushed the fix-http-inspector-request-body branch from 194a5fd to ba8f093 Compare April 30, 2026 17:33

GrinZero force-pushed the fix-http-inspector-request-body branch from aace601 to 16da9f4 Compare May 3, 2026 04:23

GrinZero force-pushed the fix-http-inspector-request-body branch from c0db72d to 7edffdf Compare May 5, 2026 10:18

GrinZero force-pushed the fix-http-inspector-request-body branch from 7edffdf to a24b943 Compare May 8, 2026 16:03

GrinZero added 2 commits May 9, 2026 14:28

inspector,http: support builtin http request bodies

8e1fea7

Signed-off-by: GrinZero <774933704@qq.com>

test: wrap inspector HTTP handler in mustCall

c3b1e4f

GrinZero added 16 commits May 9, 2026 14:28

docs: add docs for dc http client

17515d3

test: fix inspector HTTP handler call counts

8b333c1

test: inspector-network test hasPostData

af9fc23

test: cover untracked inspector HTTP body events

e607917

inspector: fix requestWillBeSent body state

66ea236

inspector: format network_agent.cc

42e26c6

test: cover network inspector initiator parsing

f14e24c

inspector,test: reject malformed initiator stack traces

23fc680

test: fix inspector protocol cctest formatting

97ce401

test: cover invalid inspector network initiator stack

5c1c8f7

inspector: document network initiator validation

910a24c

inspector: drop unreachable initiator.stack schema

e544a08

test: add more case

f43ef6e

style: add comment for createInitiator

6213aa0

inspector: tighten createInitiator coverage

dbf2c5c

style: merge split variable declaration onto one line

d0f603a

GrinZero force-pushed the fix-http-inspector-request-body branch from a24b943 to d0f603a Compare May 9, 2026 06:28

Uh oh!

Conversation

GrinZero commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Approach

1. Reuse the existing request buffering pipeline

2. Add builtin http request-body diagnostics events

3. Capture builtin http response bytes before decoding

4. Support and validate network initiator payloads

5. Document the new diagnostics channels

Behavior

Tests

Verification

Refs

Uh oh!

nodejs-github-bot commented Apr 23, 2026

Uh oh!

GrinZero commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

metcoder95 left a comment

Choose a reason for hiding this comment

Uh oh!

GrinZero commented Apr 24, 2026

Uh oh!

GrinZero commented Apr 27, 2026

Uh oh!

GrinZero commented Apr 27, 2026

Uh oh!

GrinZero commented Apr 28, 2026

Uh oh!

marco-ippolito commented Apr 28, 2026

Uh oh!

GrinZero commented Apr 28, 2026

Uh oh!

GrinZero commented Apr 28, 2026

Uh oh!

GrinZero commented Apr 28, 2026

Uh oh!

GrinZero commented Apr 29, 2026

Uh oh!

ShogunPanda left a comment

Choose a reason for hiding this comment

Uh oh!

GrinZero commented Apr 30, 2026

Uh oh!

codecov Bot commented May 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GrinZero commented May 5, 2026

Root cause

The fix

Why the C++ side has to change

How the follow-up commits fit in

Uh oh!

GrinZero commented May 6, 2026

Uh oh!

GrinZero commented May 8, 2026

Uh oh!

GrinZero commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

GrinZero commented Apr 23, 2026 •

edited

Loading

2. Add builtin `http` request-body diagnostics events

3. Capture builtin `http` response bytes before decoding

GrinZero commented Apr 23, 2026 •

edited

Loading

codecov Bot commented May 3, 2026 •

edited

Loading